Scaling Law for Recommendation Models: Towards General-Purpose User Representations

Recent advancement of large-scale pretrained models such as BERT, GPT-3, CLIP, and Gopher, has shown astonishing achievements across various task domains. Unlike vision recognition language models, studies on general-purpose user representation at scale still remain underexplored. Here we explore the possibility learning by training a universal encoder large scales. We demonstrate that scaling law is present in areas, where error scales power-law with amount computation. Our Contrastive Learning User Encoder (CLUE), optimizes task-agnostic objectives, resulting embeddings stretch our expectation what possible to do downstream tasks. CLUE also shows great transferability other domains companies, performances an online experiment significant improvements Click-Through-Rate (CTR). Furthermore, investigate how model performance influenced factors, data size, capacity, sequence length, batch size. Finally, discuss broader impacts general.